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Operational  planning  with  uncertain  and  ambiguous  information: 
Command  and  control  and  the  natural  environment 


Operational  planners,  particularly  military  planners,  are  often  faced  with  constructing  a  plan  using 
ambiguous  data  in  highly  complex  or  rapidly  evolving  situations.  Environmental  information  represents  a 
particular  challenge  for  planners.  The  state  of  the  art  in  geophysical  fluid  dynamics  leaves  significant 
uncertainty  in  forecast  conditions.  Even  with  perfect  knowledge  of  the  future  state  of  ocean  and 
atmosphere,  translating  these  conditions  into  mission  impacts  can  be  difficult  and  result  in  ambiguity  in 
interpretation.  In  this  research,  we  examine  the  use  of  meteorology  and  oceanography  (METOC) 
information  by  operational  planners.  An  experiment  was  conducted  using  human  subjects  participating  in 
a  computer-mediated  planning  simulation.  Player  teams  were  charged  with  constructing  plans  to  allocate 
assets  to  tasks  in  a  five-day  operational  scenario.  Players  were  required  to  integrate  dynamic  METOC 
information  presented  with  varying  levels  of  information  richness  (ambiguity  in  weather  conditions)  and 
varying  levels  of  information  structure  (ambiguity  in  weather  impacts).  Plans  were  evaluated  for  both 
completeness  and  robustness,  where  robustness  was  assessed  by  considering  the  plan  performance  over 
the  distribution  of  likely  METOC  conditions  in  the  mission  area.  Results  offer  insight  into  the  more 
effective  employment  of  METOC  personnel  in  the  planning  process,  and  into  better  presentation  of 
METOC  information  to  planners. 


1.  Overview 

How  do  people  make  the  best  use  of  uncertain  and  highly  perishable  information?  Operational  planners, 
particularly  military  planners,  are  often  faced  with  constructing  a  robust  and  effective  plan  using 
ambiguous  data  in  a  highly  complex  or  rapidly  evolving  situation.  Environmental  information  represents 
a  particular  challenge  for  planners.  The  state  of  the  art  in  atmospheric  and  ocean  sciences  leaves 
significant  uncertainty  in  forecast  conditions.  Even  with  perfect  knowledge,  translating  these  conditions 
into  mission  impacts  can  be  difficult;  delivering  a  more  accurate  weather  forecast  does  not  necessarily 
provide  the  planners  with  more  useful  information. 

“Listen,  S-2,  ”  the  colonel  said,  “I  don ’t  care  about  how  many  inches  of  rain  fall  to  expect.  I  don ’t  care 
about  the  percentage  of  lunar  illumination.  1  don ’t  want  lots  of  facts  and  figures.  Number  one,  1  don ’t 
have  time,  and  number  two,  they  don ’t  do  me  any  good.  What  I  need  is  to  know  what  it  all  means.  ” 

— USMC  Doctrinal  Publication  6  Command  and  Control  (1996) 

In  the  2011  Model-Based  Experiments  on  Adaptive  and  Scalable  Architectures  for  Maritime  Operations 
Centers  experiment,  we  examine  the  use  of  meteorology  and  oceanography  (METOC)  information  by 
operational  planners,  using  a  computer-mediated  planning  tool  developed  at  the  University  of  Connecticut 
(UCONN).  The  game  scenario  required  players  to  consider  METOC  information  while  assigning  assets 
to  tasks  over  a  multi-day  period.  Players  constructed  a  series  of  plans  based  on  the  resources  required  to 
complete  a  task,  the  assets  available,  and  the  effectiveness  of  those  assets  under  a  range  of  predicted 
METOC  conditions. 

Information  richness  was  manipulated  as  the  independent  variable  across  three  levels  ranging  from  low  to 
high.  Dependent  measures  examined  the  quality  of  the  plans  produced,  the  effective  use  of  assets,  and  the 
time  players  spent  in  collaboration  and  deliberation  over  different  aspects  of  the  plan.  Although  an 
obvious  hypothesis  is  that  players  given  more  and  richer  METOC  information  will  produce  better  plans, 
our  work  sought  to  develop  clear  insight  into  both  direction  and  magnitude  of  these  differences. 


2 


The  motivation  for  this  work  was,  in  part,  that  much  of  the  current  METOC  support  to  military  operations 
is  accomplished  with  automated  computer  products  and  human  weather  specialists  available  remotely  (for 
example,  by  text  chat  or  telephone)  to  deployed  forces;  this  is  operationalized  in  the  manipulation  of 
information  structure.  While  weather  and  ocean  forecasts  always  carry  some  implicit  or  ambiguous 
uncertainty  (for  example,  60%  chance  of  rain  tomorrow),  both  the  Navy  and  the  Air  Force  are  seeking 
mechanisms  to  evaluate  explicitly  the  uncertainty  attached  to  certain  numerical  weather  prediction 
techniques.  These  clear  confidence  bounds  on  the  forecast  may  improve  the  information  given  to  military 
planners  and  decision-makers,  though  it  is  not  clear  that  this  will  necessarily  lead  to  better  plans  and 
decisions.  This  explicit  uncertainty  is  operationalized  in  the  manipulation  of  information  richness,  and 
experimental  results  help  to  clarify  the  connection  between  consideration  of  uncertainty  and  improved 
decision-making. 

The  remainder  of  this  paper  is  organized  as  follows:  Section  2  provides  a  brief  background  and  review  of 
the  literature;  Section  3  presents  the  experimental  design  and  data  collection  plan;  Section  4  discusses  the 
qualitative  and  quantitative  outcomes  from  the  experiment;  and  Section  5  summarizes  the  results  and 
conclusions,  and  then  outlines  avenues  for  future  research. 

2.  Background 

2.1  The  Adaptive  Architectures  for  Command  and  Control  research  program 

The  Model-Based  Experiments  in  Adaptive  and  Scalable  Architectures  for  Maritime  Operations  Center  is 
connected  to  the  larger  Adaptive  Architectures  for  Command  and  Control  (A2C2)  program,  operating  at 
Naval  Postgraduate  School  for  over  15  years.  The  A2C2  research  program  has  developed  a  multi¬ 
disciplinary  research  agenda  to  conduct  experimentation  on  issues  critical  to  Maritime  Operations  Centers 
(MOC)  and  Maritime  Headquarters  (MHQ).  Senior  Navy  leaders  have  stated  a  need  to  refocus  and 
enhance  the  Navy’s  ability  to  function  at  the  operational  level  of  war  (U.S.  Fleet  Forces  Command,  2007). 
MOCs  were  conceived  to  enable  these  capabilities  while  providing  a  degree  of  standardization  among  the 
maritime  headquarters  (MHQ  with  MOC  Concept  of  Operations,  2007).  A  MOC  empirical  research 
campaign  is  underway  where  the  emphasis  is  on  operational  versus  tactical  activities,  and  planning 
versus  reacting.  Because  of  its  complexity,  its  mission  to  oversee  large  operations,  and  its  dynamic 
structure,  the  MOC  is  an  ideal  organization  for  research  on  organizational  structure,  C2,  and  the  process 
of  mission  planning.  Since  the  MOC  was  designed  to  effectively  integrate  the  planning  elements  of 
Current  and  Future  Operations  (COPS  and  FOPS)  to  provide  more  rapid  and  accurate  resource  allocations 
that  are  consistent  with  mission  requirements,  our  first  experiment  focused  on  the  MOC  with  emphasis  on 
intelligence,  surveillance,  and  reconnaissance  (1SR)  (Hutchins  et  al.,  2009). 

A  second  experiment  conducted  in  2010  to  investigate  an  issue  relevant  to  MOCs.  In  this  study,  the 
MOC  planning  teams  operated  in  either  an  integrated  or  isolated  fashion.  The  integrated  FOPS  team  was 
supported  by  a  decision  aid  and  planning  tool  that  fostered  coordination,  while  the  isolated  FOPS  team 
used  a  planning  tool  with  a  reduced  coordination  capability.  The  objective  of  this  experiment  was  to 
examine  the  potential  problems  that  could  arise  when  forming  an  operational  planning  team  (OPT).  An 
OPT  is  a  task-organized  team  formed  to  conduct  integrated  planning  for  a  specific  mission.  An  OPT  is 
often  formed  by  the  MOC  because  it  offers  the  advantage  of  a  focused  group  of  subject  matter  experts 
approaching  the  problem  in  an  integrated  manner.  However,  problems  may  be  associated  with  this  team 
being  somewhat  isolated  in  situations  that  require  the  OPT  to  coordinate  closely  with  the  rest  of  the 
MOC.  The  overarching  research  question  sought  to  understand  how  emergent  events  are  best  handled 
when  resources  must  be  shared  among  separate  planning  teams— for  example  when  an  OPT  is  formed. 
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The  2010  study  was  designed  to  examine  the  efficiency  and  planning  performance  of  two  alternative 
organizational  structures:  (1)  Integrated  -  where  planning  teams  plan  with  a  real-time  view  of  others’ 
resource  planning,  and  (2)  Isolated  -  where  planning  teams  operate  in  isolation,  without  the  ability  to 
directly  view  others’  resource  planning.  The  first  experimental  hypothesis  was  that  “integrated  teams 
create  more  effective  plans  than  isolated  teams  due  to  their  real-time  awareness  that  enhances  the 
interdependent  solution.  Our  second  experimental  hypothesis  was  that  isolated  team  members  experience 
higher  levels  of  workload  than  integrated  team  members  because  their  lack  of  real-time  planning  status 
requires  more  frequent  status-related  communication  in  addition  to  collaborative  effort.  Our  third 
experimental  hypothesis  was  that  isolated  team  members  communicate  more  frequently  in  response  to 
emergent  events  because  isolated  team  members  must  communicate  to  learn  how  others  alter  plans  in 
response  to  unexpected  events. 

Teams  in  the  integrated  condition  produced  higher  quality  plans  than  the  isolated  condition  due  to  the 
enhanced  shared  situation  awareness  provided  by  the  planning  software  and  as  expected,  the  average 
overall  workload  reported  by  isolated  team  members  was  significantly  greater  than  workload  reported  by 
integrated  team  members  across  all  four  experimental  sessions  (Hutchins  et  al.,  2010).  The  lack  of  shared 
situation  awareness  provided  by  their  planning  tool  necessitated  an  increase  in  the  amount  of  explicit 
(communication-based)  status  updates  and  coordination  required  to  succeed  on  this  interdependent  task. 
Results  suggest  that  as  the  scenario  builds  and  the  interdependence  between  task  areas  A  and  B  became 
more  complex,  integrated  team  members  perceived  less  mental  workload  and  demonstrated  more 
effective  performance  than  isolated  team  members. 

The  objectives  for  the  present  study  are  twofold,  to:  (1)  continue  our  model-based  experimentation,  and 
(2)  explore  new  paradigms  for  empirical  studies  of  critical  issues  appropriate  for  MOC  laboratory 
research.  Through  the  integration  of  analytical  modeling,  human-in-the-loop  experimentation  and 
computer  simulation,  our  research  has  followed  a  “model-test-model-experiment”  paradigm  wherein 
models  and  associated  simulations  define  and  guide  the  experiments,  and  the  results  from  the  experiments 
are  fed  back  to  improve  and  enhance  the  models.  In  previous  research  we  have  focused  on  resource 
allocation  in  planning  under  different  collaborative  structures  for  sharing  information.  An  implicit 
assumption  in  these  studies  was  that  this  shared  information  was  “perfect.”  In  the  present  study  we 
examine  the  use  of  imperfect,  mission-critical  information:  characterization  of  the  natural  environment. 

2.2  The  ambiguity  of  forecasts  for  the  atmosphere  and  oceans 

Weather  prediction  is  appealing  as  a  purely  deterministic,  physical  problem  (Regnier,  2008b).  This 
deterministic  approach,  however,  is  limited  by  physical  realities  in  how  well  we  can  characterize  the 
initial  state  of  the  atmosphere  and  oceans,  and  the  degree  to  which  we  can  carry  this  information  through 
our  time  integration  of  the  coupled  differential  equations  that  describe  the  atmosphere  (Lorenz,  1963). 
This  deterministic  approach  yields  a  single,  physically  plausible  solution,  though  the  accuracy  of  this 
solution  begins  to  deteriorate  quickly  after  about  144  hours  in  to  the  forecast  under  the  best  of  conditions. 
Put  simply,  even  with  the  technology  and  science  we  bring  to  bear  in  the  21st  century,  we  are  still 
challenged  by  accurate  weather  predictions  beyond  about  six  days. 

Every  numerical  weather  prediction  carries  an  inherent  uncertainty.  Typically,  the  bounds  of  this 
uncertainty  are  unknown,  though  a  skilled  human  weather  forecaster  can  often  interpret  model  output  to 
attach  a  reasonable  estimate  of  the  uncertainty.  Although  this  uncertainty  estimate  may  be  qualitative 
(e.g.,  the  model  often  over-predicts  rainfall  amounts  in  the  mountains),  the  human  forecaster  nonetheless 
can  make  this  uncertainty  clearer  to  decision-makers  (Doswell,  2004). 

Another  approach  to  evaluating  model  uncertainty  is  to  compare  several  models  over  the  same  forecast 
period,  examining  the  mean  and  variance  of  the  ensemble  solution  (Leutbecher  &  Palmer,  2008). 
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Regions  of  high  variance  in  the  ensemble  may  be  interpreted  as  areas  of  significant  uncertainty.  The 
explicit  numbers  derivable  from  ensemble  output  provide  another  means  to  make  clear  to  decision-makers 
how  much  trust  to  place  in  a  particular  forecast  for  a  particular  region.  The  interpretation  of  these 
numbers,  however,  is  not  always  clear  to  decision-makers  and  may  require  some  intervention  by  human 
weather  forecasters  (Morss,  2010). 

2.3  Creating  actionable  intelligence  from  ambiguous  predictions 

No  matter  the  skill  with  which  we  predict  the  future  state  of  the  atmosphere  and  oceans,  these  predictions 
are  of  themselves  of  little  use  to  military  planners.  Decision-makers  are  largely  concerned  with  when  and 
to  what  degree  their  assets  and  capabilities  will  be  affected  by  weather  conditions  (Moore  et  al.,  2003). 
For  planners  with  a  trade-space  spanning  days  or  weeks,  decisions  to  proceed  with,  accelerate  or  delay 
operations  are  connected  to  the  expected  weather  conditions. 

While  at  the  tactical  level  it  is  common  for  operators  to  simply  avoid  or  react  to  the  weather,  at  the 
operational  level  planners  may  employ  environmental  forecasts  to  help  ensure  the  best  use  of  assets  in 
time  and  space.  Assuming  that  bad  weather  is  transient,  and  particularly  if  the  area  of  responsibility 
covers  multiple  weather  regimes,  planners  can  adjust  assignments  in  concert  with  forecast  environmental 
conditions.  In  bad  weather  areas,  planners  can  often  delay  operations  and  reassign  assets  to  operations  in 
relatively  good  weather  areas,  increasing  effort  where  the  environment  permits.  These  deferred  tasks  in 
initially  bad  weather  areas  may  then  receive  increased  effort  when  the  weather  improves.  Effective 
planning  in  this  case  is  connected  to  knowledge  and  exploitation  of  environmental  forecasts. 

2.4  Summary 

One  may  argue  that  all  planning  takes  place  under  uncertainty.  The  motivation  for  this  work  is  that  the 
natural  environment  and  its  inherent  uncertainty  represent  both  challenges  and  opportunities  for  the 
operational  planner.  The  degree  to  which  this  information  is  useful  to  the  planner,  however,  may  depend 
upon  the  information  content  and  structure.  We  next  describe  the  research  design  for  this  investigation. 

3.  Method 

3.1  Research  question  and  working  hypothesis 

We  seek  to  better  understand  how  organizations  employ  perishable  and  uncertain  information  in  the 
operational  planning  process.  Does  providing  more  information  to  planners  lead  to  better  planning?  In 
the  face  of  inherently  uncertain  information,  if  we  make  this  uncertainty  more  clear  through  human 
intercession,  or  explicit  quantitative  bounds,  will  planners  use  and  integrate  this  information  in  their 
deliberations? 

This  research  is  a  natural  evolution  from  previous  A2C2  studies  involving  integrated  planning  teams  in 
the  US  Navy  Maritime  Operations  Center  (Hutchins  et  al.,  2009),  most  recently  under  different 
collaborative  structures  (Hutchins  et  al.,  2010).  These  efforts  were  distinguished  by  computer-mediated 
experiments  with  human  subjects  playing  the  role  of  planning  teams  supporting  the  MOC.  In  the  present 
study  we  examine  the  MOC  planning  process  under  a  scenario  where  player  actions  are  constrained  by 
events  in  the  natural  environment  (e.g.,  thunderstorms,  heavy  seas,  dense  cloud  cover).  Given  the  same 
environmental  scenario,  but  markedly  different  information  about  the  uncertainty  in  the  natural 
environment,  we  ask:  do  teams  integrate  and  apply  this  information  effectively  in  their  mission  plans? 

If  we  think  of  the  experimental  levels  as  ranging  from  low  to  high  in  terms  of  uncertainty  information,  we 
hypothesize  that  planning  teams  given  high  (richer)  uncertainty  information  will  outperform  teams  given 
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low  (less  rich)  uncertainty  information.  While  this  may  seem  obvious,  our  design  is  intended  to  address 
the  perhaps  more  useful  question:  how  much  better  do  teams  perform  given  richer  uncertainty 
information? 

This  question  is  of  significant  operational  relevance  to  both  the  Navy  and  Air  Force,  as  there  is  a  cost  to 
keep  humans  deeply  embedded  in  the  forecast  process,  and  a  cost  to  produce  explicit  uncertainty  bounds 
with  numerical  forecasts.  Within  the  Department  of  Defense  (DoD)  the  current  trend  is  to  consolidate 
METOC  personnel  in  centers,  typically  located  far  from  the  forward  edge  of  battle,  and  most  often 
located  in  the  continental  United  States  (CONUS).  Support  to  deployed  operations  is  then  provided  with 
online  product  delivery  and  reach  back  service  to  these  centers.  Both  the  Navy  (Sestak  et  al.,  2008)  and 
the  Air  Force  (Nobis  et  al.,  2008)  are  examining  the  use  of  ensemble  numerical  weather  prediction  to 
improve  operational  forecasts  and  improve  the  explicit  uncertainty  information  attached  to  these 
forecasts.  For  both  services,  though,  a  lingering  concern  is  whether  forecasters  and  decision-makers  will 
correctly  and  effectively  employ  the  richer  ensemble  information.  Although  the  present  study  is  not  about 
valuing  ensemble  forecasting,  the  insights  into  the  way  planners  apply  the  uncertainty  in  environmental 
information  may  prove  useful  to  Navy  and  Air  Force  weather  organizations. 

3.2  Experiment  participants 

Twenty-four  students  in  the  Graduate  School  of  Operational  and  Information  Sciences,  at  the  Naval 
Postgraduate  School  (NPS),  Monterey,  CA,  served  as  experimental  participants  during  a  ten-hour 
experiment  conducted  28  February-1 1  March  2011.  Their  mean  age  was  33.4  years;  services  represented 
were  Navy,  17,  Marine  Coips,  6,  and  Army,  1.  Participants’  rank  ranged  from  0-2  to  0-6.  Six  teams  of 
four  players  participated  in  the  experiment,  with  each  playing  the  role  of  a  Future  Operations  (FOPS) 
planning  team  under  one  of  three  experimental  conditions. 

3.3  Experiment  overview 

Similar  to  previous  A2C2  experiments  (Flutchins  et  al.,  2009;  2010),  teams  were  comprised  of  four 
players  operating  with  a  shared  goal  while  working  together  over  four  two-hour  sessions.  Within  the 
team,  each  player  was  assigned  responsibility  for  a  region  (Area  A  or  Area  B)  and  planning  period.  In  the 
four-day  scenario,  the  current  day  was  always  designated  as  T;  the  plan  for  the  next  day  was  designated 
T+l;  and,  the  plan  48  hours  out  was  the  T+2  plan.  Players  worked  on  plans  in  a  rolling  horizon,  so  that 
the  current  T+2  plan  would  be  the  starting  point  for  T+l  plan  developed  tomorrow,  ideally  reflected  in 
within-team  coordination  between  the  T+l  and  T+2  planners  for  Areas  A  and  B.  Each  player  was  given  a 
weather  forecast  consistent  with  their  area  of  responsibility  within  the  theater,  and  with  their  planning 
period.  The  richness  of  this  forecast  information  was  varied  by  experimental  condition. 

Each  player  was  also  presented  with  a  set  of  tasks  to  accomplish  in  their  respective  areas  of  responsibility 
and  within  the  four  play  sessions.  These  tasks  were  oriented  to  the  geography  of  the  theater  (Areas  A  and 
B),  so  that  a  team-within-team  structure  was  enforced,  with  two  players  working  on  Area  A  tasks  for  T+l 
and  T+2,  and  similarly  two  players  working  on  Area  B  tasks.  All  tasks  were  accomplished  with  shared 
(and  limited)  resources  among  the  team  members,  which  drove  collaboration  across  geographic  areas 
between  A  and  B  teams.  Inside  the  team-within-team  structure,  the  shared  tasks  for  an  area  required 
coordination  between  T+l  and  T+2  players.  For  example,  the  Area  B  team  was  given  a  set  of  tasks  to 
accomplish  within  the  four  sessions,  though  in  each  session  the  T+2  player  needed  to  know  how  much  the 
T+l  player  could  accomplish  in  that  planning  period  prior  to  building  the  T+2  plan. 

Each  player  was  given  a  weather  forecast  as  a  starting  point  for  his  planning  work  during  the  session, 
consistent  with  his  area  of  responsibility  (A  or  B),  planning  period  (T+l  or  T+2),  and  the  team 
experimental  condition  in  information  richness.  The  experimental  scenario  was  constructed  in  such  a  way 
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as  to  create  significant  weather  impacts  and  changes  over  the  four  sessions  (Figure  1),  though  these  events 
were  phased  so  that  in  general  if  weather  were  good  in  one  area  it  would  be  poor  in  the  other  area.  While 
this  weather  phasing  may  seem  contrived  for  the  purpose  of  the  experiment,  it  is  worth  noting  that  in  large 
areas  of  responsibility  (for  example,  in  US  Central  Command  and  Southwest  Asia)  this  is  consistent  with 
the  observed  weather  pattern  over  a  two  to  five-day  period. 

We  hypothesize  that  planning  teams  with  richer  information  will  plan  more  effectively.  In  response  to 
forecast  weather  conditions,  we  expect  teams  to  (1)  defer  tasks  until  later  in  the  period,  when  the  weather 
is  presently  bad  but  forecast  to  improve;  (2)  accelerate  tasks  beyond  normal  workload,  when  the  weather 
is  presently  good  but  forecast  to  deteriorate;  and,  (3)  coordinate  labor  among  areas  of  responsibilities  so 
that  tasks  in  fair  weather  regions  can  receive  more  attention  to  more  effectively  use  resources  within  a  24- 
hour  period  in  the  experiment  scenario.  The  experimental  scenario  was  designed  to  clearly  give  players 
this  trade  space,  while  the  dependent  measures  were  intended  to  evaluate  how  well  players  operated  in 
this  trade  space  given  different  levels  of  information  richness. 
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Figure  1.  The  experiment  scenario  was  crafted  over  imaginary  geography,  with  a  challenging  weather  scenario 
affecting  the  tasks  in  Areas  A  and  B.  Task  forces  (TF  A  through  E)  are  indicated  as  well. 


3.4  Experiment  design:  Player  planning  tasks 

All  players  completed  their  experimental  work  using  a  computer-mediated  future  operations  (FOPS) 
planning  tool  developed  at  the  UCONN  (Mandal  et  al.,  2010;  Flan  et  al.,  2010).  This  tool  served  as  the 
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primary  means  of  data  collection  during  the  experiment;  all  of  the  players’  deliberations  and  final 
decisions  were  recorded  for  post-experiment  analysis.  Equally  important,  this  tool  served  as  a 
fundamental  means  to  operationalize  the  FOPS  planning  process.  In  modeling  the  work  of  planners  at  the 
operational  level,  players  made  assignments  at  the  task  force  level,  while  considering  the  tactical 
requirements  for  each  assigned  task  and  the  tactical  capabilities  of  each  task  force.  The  modeling 
formalism  used  to  match  assets  to  tasks  is  depicted  in  Figure  2. 

Within  the  FOPS  tool,  players  used  the  Assignment  page  (Figure  3)  to  reflect  their  chosen  assignments  of 
task  forces  to  tasks.  In  game  play,  assigning  a  task  involved:  (a)  selecting  a  primary  task  force,  and  (b) 
selecting  up  to  two  supporting  task  forces  if  needed  to  meet  the  warfare  requirements  of  a  task  (Figure  3). 
For  example,  in  Figure  3,  the  player  is  working  on  task  TA02  (conduct  intelligence,  surveillance  and 
reconnaissance  on  ground  targets  in  Area  A)  that  requires  32  units  of  the  capability  ISRg,  16  units  of 
capability  C2  and  16  units  of  capability  ISRs.  The  player  has  selected  Task  Force  A  (TF-A)  as  the 
primary  task  force  for  this  task. 

In  addition  to  requesting  a  task  force  (or  task  forces)  for  the  task  at  hand,  the  planner  may  adjust  the 
desired  percent  complete  (Figure  3,  dashed  oval  at  top  center)  to  better  fit  the  available  resources.  The 
intent  was  for  players  to  meet  their  desired  percent  complete  on  a  task  within  the  24-hour  (T+l)  planning 
cycle  without  requesting  more  effort  than  available  (because  of  weather  or  task  force  limitations)  and 
without  underestimating  the  possible  effort  achievable  with  the  forecast  weather  conditions.  Dependent 
measures  in  the  experiment  captured  this  asset  efficiency  as  one  measure  of  effective  planning. 


MODELING  FORMALISM:  MATCHING  TASKS-to-ASSETS 


Task  Requirements  TF  Capabilities 


TASKS  AND  ASSETS  ARE 
REFERENCED  TO  THE 
SAME  SET  OF  SELECTED 
WARFARE  CATEGORIES 

•  Ri!c:  Requirement  /coftask  / 

•  Capability  A  of  asset/TF; 


Values  (resource  vectors)  are 
readily  obtainedvia  discussions 
withSMEsor  Fleet  personnel 


TF  CAPABILITIES  ARE  DAY-by-DAY  ROLL-UPS  OF  THE  INDIVIDUAL  ASSET  CAPABILITIES 


•  FOPS  ASSIGNS  EACH  TASK  TO  A  TF,  with  PERFORMANCE  GOALS  and  PRIORITIES 

-  One  primary  TF,  with  others  assigned  as  supporting  in  one  or  more  warfare  areas 

-  The  TFs  determine  how  to  best  utilize  their  assets  to  meet  performance  goals 


Figure  2.  In  the  modeling  formalism  within  this  experiment,  players  assign  Task  Forces  (comprised  of  multiple 
warfare  areas  or  assets)  to  tasks.  A  limit  on  the  number  of  tasks  to  which  a  Task  Force  may  be  assigned  helps  to 
create  tension  in  the  players’  trade  space. 


The  task  force  composition  and  warfare  capabilities  were  developed  as  part  of  the  experiment  scenario. 
Relative  numbers  (e.g.  50  units  of  C2)  were  developed  from  discussions  with  subject  matter  experts. 
Clearly,  resource  scarcity  could  be  engineered  from  this  perspective,  giving  players  only  a  few  options  to 
arrive  at  a  “correct”  solution;  this  approach  was  used  in  previous  experiments  (Hutchins  et  al.,  2010).  In 
order  to  drive  planning  teams  to  employ  environmental  information  in  their  deliberations,  however,  the 
resources  as  assigned  initially  were  chosen  to  make  the  four-session  scenario  relatively  easy  to  complete 
under  perfect  weather.  Resource  scarcity,  then,  was  almost  solely  a  function  of  weather  impacts  in  Areas 
A  and  B  (Figure  1). 


Figure  3.  The  Assignment  page  from  the  FOPS  planning  tool  shows  the  task  being  worked  (dashed  oval  in  the 
upper  left  of  the  figure)  and  the  task  force  assignments.  The  player's  selected  desired  percent  complete  is  indicated 
by  the  dashed  oval  at  top  center. 

3.5  Experiment  design:  Operationalizing  weather  impacts 

The  Weather  page  (Figure  4)  and  Assignment  page  (Figure  3)  were  used  in  concert  by  planning  teams  to 
coordinate  on  tasks  and  resources  within  a  planning  session.  The  Weather  page  presented  both  forecast 
weather  (controlled  by  the  scenario)  and  a  what-if  assumed  weather  condition  (Figure  4).  To  simplify  the 
environmental  information  within  the  experiment,  the  players  were  presented  with  only  three  weather 
parameters:  sea  condition,  cloud  cover,  and  thunderstorms.  These  weather  parameters  were  each 
expressed  on  a  scale  from  0  to  100,  where  100  indicates  perfect  weather  with  no  impact  to  warfare  areas 
(Figure  5).  Although  players  were  not  given  the  weather  curves  for  the  scenario  warfare  areas  (Figure  5), 
the  range  of  expected  forecast  impacts  were  indicated  with  a  simple  Assumed  +5/-5  evaluation  (Figure  4, 
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center).  This  range  is  based  on  the  players’  choice  of  assumed  weather  and  shifted  as  players  engaged  in 
what-if  scenarios  around  the  forecast  weather  conditions. 

Given  a  task  in  a  bad-weather  region,  players  coidd  opt  to  defer  the  task  to  later  in  the  four  sessions,  or 
attempt  task  completion  by  applying  additional  assets  in  warfare  areas  affected  by  the  weather  conditions. 
In  the  limited  assets  available  to  the  team,  however,  this  would  require  coordination  among  the  Area  A 
and  B  teams-within-team. 
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Figure  4.  The  Weather  page  provides  planners  with  forecast  weather  conditions,  along  with  additional  uncertainty 
and  integrative  information  per  the  experimental  condition.  This  example  is  for  Level  III,  with  explicit  uncertainty 
bounds  (center  dashed  oval)  and  integrative  information  regarding  the  tasks  (bottom  dashed  oval). 

The  independent  variables  were  operationalized  on  the  Weather  page,  with  low-richness  players  operating 
with  no  forecast  bounds  or  integrative  mission  information  (indicated  in  Figure  4).  With  moderate 
information  richness,  players  were  given  the  explicit  uncertainty  bounds  (dashed  oval,  center  of  Figure  4), 
though  planning  teams  still  needed  to  use  the  what-if  tool  to  examine  the  possible  weather  impacts.  In  the 
high-richness  condition,  planning  teams  were  given  both  uncertainty  bounds  and  integrative  information 
(Figure  4).  Because  the  what-if  tool  was  available  in  all  experimental  conditions,  the  number  of  times 
players  employed  this  tool  represented  another  useful  dependent  measure  to  assess  players’  awareness 
and  application  of  information  about  the  natural  environment. 

3.6  Experiment  execution 

The  experiment  was  conducted  in  four  two-hour  time  blocks,  spread  over  a  two-week  period.  The  MOC 
Director  (played  by  a  confederate)  presented  the  Commander’s  Update  Briefing  at  the  beginning  of  each 
experimental  session.  This  briefing  reviewed  the  scenario  with  players  and  updated  teams  on  any  changes 
in  time-phased  force  deployment  data  (TPFDD)  and  resources  available  to  the  players.  The  MOC 
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Director  also  reviewed  the  task  graph  with  the  players  to  give  an  informal  assessment  of  team  progress  on 
assigned  tasks.  A  sample  task  graph  is  depicted  in  Figure  6. 

Block  0  consisted  of  an  introduction  to  the  experiment,  including  a  brief  on  the  mission,  and  initial 
training  in  their  team  roles.  Players  also  practiced  with  the  computer-mediated  FOPS  tool. 

Block  1  was  the  first  full  session  where  players  were  given  the  initial  state  of  the  scenario  and  tasked  with 
building  plans  for  Block  2  (T+l)  and  Block  3  (T+2).  FOPS  obtained  updated  information  throughout  the 
session  to  include  updates  to  the  weather  forecasts  for  T+l  and  T+2.  The  FOPS  planning  update  for  T+l 
and  T+2  (Blocks  2  and  3)  was  briefed  to  the  MOC  Director  and  submitted  as  a  new  plan  at  the  end  of  this 
block. 


Figure  5.  Response  curves  relating  forecast  cloud  condition  (on  a  scale  of  0  to  100)  to  expected  asset  effectiveness 
(on  a  scale  of  0  to  1).  For  the  purpose  of  the  experiment,  the  forecast  condition  closer  to  100  means  better  weather 
and  less  impact  to  asset  effectiveness.  In  this  case,  cloud  cover  effects  strike  (STRK),  ground  reconnaissance 
(ISR  g),  and  battle  damage  assessment  (BDA). 


Block  2  was  the  second  session,  beginning  with  the  implementation  of  the  FOPS  team  plan  produced  in 
Block  1.  The  FOPS  team  created  a  plan  for  Block  3  using  their  Block  1  plan  for  T+2  as  a  first  guess. 
This  team  also  created  the  Block  4  (T+2)  plan  based  on  their  updated  Block  3  (T+l)  plan  and  previous 
progress. 

Block  3  was  the  third  session,  and  FOPS  teams  implemented  the  Block  2  plan  for  T+l,  and  used  the 
Block  2  T+2  plan  as  the  starting  point  for  the  T+l  (Block  4)  plan.  In  this  session  the  FOPS  team  also 
created  a  T+2  (Block  5)  plan,  although  no  such  block  was  executed  in  this  experiment.  A  briefing  with 
the  MOC  Director  was  held  at  the  end  of  Block  3. 

Block  4  was  the  final  session  of  the  experiment.  This  session  implemented  the  Block  3  plan  for  T+l,  and 
player  teams  used  the  Block  3  T+2  plan  as  the  starting  point  for  the  T+l  (Block  5)  plan.  A  T+2  (Block  6) 
plan  was  also  created  based  on  the  actual  progress  to  date,  and  the  expected  progress  of  the  T+l  (Block  5) 
plan.  Neither  of  these  plans  for  Block  5  or  6  (T+l  or  T+2)  is  executed  in  the  experiment  scenario. 
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Area  A  Tasks 


Figure  6.  Sample  task  graph  for  Area  A  players.  Dashed  ellipses  indicate  the  planning  window  for  block  and  day, 
where  Day  0  plans  Block  0.  Note  that  Block  0  was  a  training  session  and  all  players  were  given  a  notional  plan  at 
the  end  of  the  session  so  that  all  teams  started  Block  1  with  the  same  initial  conditions. 

Within  the  four  sessions,  there  are  three  opportunities  to  examine  overlapping  plans  as  a  dependent 
measure.  For  example,  the  T+2  plan  for  Block  2  becomes  the  initial  guess  for  the  Block  3  T+l  plan.  This 
plan  volatility  provides  a  useful  insight  into  the  effectiveness  of  the  planning  process  for  the  team. 

In  addition  to  dependent  measures  collected  within  the  FOPS  planning  tool,  players  were  also  asked  to 
brief  their  plans,  to  include  their  rationale  for  task  assignments.  This  permitted  subjective  data  collection 
with  additional  insight  into  how  the  players  across  experimental  conditions  chose  to  integrate  weather  into 
their  planning  processes. 

3.7  Independent  variables 

Viewed  as  an  information  processing  entity,  the  organization  seeks  information  to  reduce  uncertainty, 
with  adjustments  in  collaborative  and  coordinating  structures  tuned  to  better  accomplish  this  goal 
(Galbraith,  1974).  In  the  present  work,  we  do  not  modify  the  collaborative  structure  but  instead  examine 
the  changes  in  media  richness  (Daft  &  Lengel,  1986)  for  effect  on  the  information  processing  of  planning 
teams  within  a  modeled  Maritime  Operations  Center. 

The  levels  of  the  independent  variable  as  originally  conceived  separated  content  and  structure  of  the 
weather  forecast  products  delivered  into  a  classic  2X2  design  (Table  1). 
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Information  Content  (a,  b) 

Forecast  Only 

Forecast  with 
Explicit  Uncertainty 

Information 

Structure 

(1,  ID 

Automated 

Products 

Level  la 

Level  lb 

Automated 
Products  with 
Human  Expertise 

Level  Ha 

Level  lib 

Table  1.  Original  experimental  concept  for  levels  of  the  independent  variables. 

In  this  approach,  Levels  la  and  11a  represent  much  of  the  current  practice  in  the  Navy,  Air  Force  and 
National  Weather  Service  (Morss,  2010).  Operationally,  most  atmosphere  and  ocean  products  are 
presented  as  deterministic  forecasts  with  implicit  uncertainty  (la).  This  uncertainty  is  often  clarified  by 
additional  information  from  experienced  human  forecasters  (lla).  Analysis  of  these  levels  would  yield 
some  insight  into  the  substantive  contribution  by  human  forecasters  to  the  planning  process. 

Manipulation  of  information  content  would  make  explicit  the  uncertainty  attached  to  the  weather  forecast 
(Levels  lb  and  lib).  This  could  be  examined  with  (lib)  and  without  (lb)  human  clarification,  though  the 
current  literature  suggests  that  correct  interpretation  of  explicit  uncertainty  in  environmental  products 
would  require  some  further  expertise  by  weather  specialists  (Doswell,  2004).  While  this  independent 
variable  may  still  be  interesting,  it  is  possible  that  Level  lib  could  muddle  the  notion  of  the  weather 
specialist  bringing  to  bear  meteorological  expertise  and  mission-focused  expertise.  Consider,  for 
example,  a  case  where  we  know  that  a  particular  aviation  platform  requires  surface  winds  below  10 
nautical  miles  per  hour  (knots,  or  kts)  to  take  off,  and  the  forecast  for  tomorrow  is  8  kts  with  a  standard 
deviation  of  3  kts.  Our  meteorological  expert  could  offer  that  (1)  the  forecast  cross  winds  will  be  8  kts 
with  a  good  chance  of  gusts  to  11  kts;  or,  (2)  forecast  cross  winds  tomorrow  have  a  good  chance  of 
violating  the  take-off  threshold  for  our  aviation  platform,  with  a  mean  of  8  kts  but  a  strong  possibility  of 
1 1  kts  in  the  period.  Cross  winds,  here,  refers  to  the  wind  component  oriented  90  degrees  across  the 
runway.  We  assert  that  this  additional,  integrative  information  relating  weather  to  mission  impact  is  itself 
an  interesting  independent  variable,  though  a  2X3  experimental  design  becomes  difficult  to  manage. 

To  better  meet  our  experimental  objectives  we  chose  to  examine  both  aspects  of  the  weather  specialists’ 
contribution  to  information  processing  using  a  compromise  design.  We  transformed  the  2X2  structure 
into  a  1X3  approach,  collapsing  information  structure  and  content  into  a  single  independent  variable, 
information  richness,  as  depicted  in  Table  2. 


Level  I 

Level  11 

Level  III 

Information 

Richness 

Forecast  only,  no 
additional  weather  or 
mission  information 

Forecast  with  explicit 
uncertainty  information 

Forecast  with  explicit 
uncertainty  and  integrating 
mission  information 

Table  2.  Adjusted  design  for  the  2011  experiment. 

In  this  approach,  teams  in  Level  1  were  presented  with  a  simple  weather  forecast  relevant  to  their  planning 
activities  but  without  explicit  uncertainty  nor  with  integrating  mission-impact  information  (human 
expertise);  this  condition  represents  the  control  for  Levels  II  and  III.  In  Level  II,  the  weather  forecasts 
were  provided  to  teams  with  explicit  uncertainty  information  (e.g.,  winds  tomorrow  will  be  8  kts  plus  or 
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minus  3  kts),  with  additional  expertise  available  only  with  respect  to  the  forecast.  In  Level  111,  weather 
forecasts  with  uncertainty  and  integrating,  mission-specific  impact  assessments  were  provided  to  teams 
for  their  planning  activities. 

In  terms  of  information  richness,  we  characterize  Level  I  as  low  and  Level  III  as  high.  While  we  expect 
team  performance  to  improve  from  Level  I  to  Level  III,  the  magnitude  of  these  differences,  and  the 
distinctions  between  Levels  II  and  III,  are  the  focus  of  our  experiment  and  analysis.  We  next  discuss  the 
experimental  measures  used  to  examine  these  differences. 

3.8  Dependent  measures 

As  information  richness  increases  from  low  to  high,  we  hypothesize  that  planning  teams  given  richer 
environmental  information  will  perform  better.  The  comprehensive  data  collection  within  the  FOPS 
planning  tool  permitted  examination  of  several  dependent  variables  to  quantify  performance.  Broadly, 
these  can  be  thought  of  as:  completion  measures;  efficiency  measures;  and,  counting  measures. 

Completion  measures  were  used  to  evaluate  team  performance  in  completing  the  assigned  tasks  within 
their  task  graph  (see,  for  example,  Figure  6).  The  overall  task  completion  at  the  end  of  the  four  sessions 
offered  an  absolute  completion  measure  and  an  overall  means  of  comparing  teams  across  the  levels  of  the 
independent  variable.  We  hypothesize  that  teams  in  Level  III  (integrative  weather  and  mission  impact 
information)  will  outperform  teams  in  Levels  II  and  I. 

The  task  progress  per  session  represented  an  excellent  mechanism  for  immediate  team  feedback.  That  is, 
at  the  beginning  of  the  session,  players  were  briefed  on  their  relative  progress  in  the  past  session  based  on 
observed  weather  conditions  and  task  force  assignments.  Because  the  players’  trade  space  spans  several 
sessions,  this  per-session  measure  is  likely  not  as  useful  for  evaluating  team  performance. 

Efficiency  measures  provided  insight  into  how  well  a  team  applied  assets  to  tasks.  Asset  efficiency  is 
maximized  when  assets  are  applied  to  tasks  in  relatively  good  weather;  asset  efficiency  is  minimized 
when  players  attempted  to  accomplish  tasks  in  bad  weather  areas.  This  measure  indicates  how  well  a 
team  adjusted  to  bad  weather  by  shifting  work  from  one  area  to  the  other.  We  hypothesize  that  teams  in 
Level  II  and  III  will  make  more  efficient  use  of  assets  than  teams  in  Level  I.  We  expect,  too,  that  Level 
III  will  be  more  efficient  than  Level  II. 

Counting  measures  reflect  the  number  of  times  something  happened  in  game  play.  In  particular,  we 
assessed  the  players’  belief  in  the  weather  forecast  by  examining  the  number  of  times  the  player  used  the 
what-if  tool  to  change  weather  parameters.  Each  of  the  four  members  of  a  planning  team  had  access  to 
the  weather  what-if  mechanism  for  their  planning  window  (T+l  or  T+2)  and  area  (Area  A  or  B),  so  this 
measure  could  be  evaluated  at  both  the  level  of  the  player  and  the  team.  At  the  team  unit  of  analysis,  we 
hypothesize  that  players  with  richer  information  will  spend  more  time  exploring  weather  and  mission 
impacts,  while  players  with  less  rich  information  will  be  more  likely  to  simply  accept  the  forecast  weather 
conditions  as  given. 

Another  useful  counting  measure  is  the  change  of  primary  task  force  between  overlapping  plans.  Blocks 
2,  3,  and  4  offered  overlapping  plans  for  the  T+l  player,  who  started  with  the  T+2  plan  from  the  previous 
day  as  an  initial  plan.  The  changes  between  the  final  T+l  plan  and  the  initial  (previous  day’s  T+2)  plan 
represent  a  measure  of  volatility  and  give  some  insight  into  the  robustness  of  the  team  planning  process. 
Less  volatility  represents  a  more  robust  plan,  and  we  hypothesize  that  teams  with  richer  information  will 
construct  less  volatile  (more  robust)  plans. 
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3.9  Summary 


Does  providing  more  information  to  planners  improve  the  planning  process?  Using  a  scenario  driven  by 
environmental  impacts  and  modeled  on  MOC  planning  tasks,  we  investigated  this  question  with  a 
computer-mediated  team-in-the-loop  experiment.  Information  richness  was  manipulated  as  the 
independent  variable  across  three  levels  ranging  from  low  to  high.  Dependent  measures  of  the  planning 
process  include  assessments  of  asset  efficiency;  absolute  and  complete  progress  in  assigned  tasks;  plan 
volatility  between  overlapping  T+l  and  T+2  plans;  and  subjective  measures  from  post-session  player 
briefings  and  surveys.  We  next  present  an  analysis  of  the  experimental  outcomes. 

4.  Results 

The  experiment  was  conducted  in  late  February  and  early  March  2011  at  the  Naval  Postgraduate  School 
(NPS),  Monterey,  California.  In  this  section  we  summarize  the  experiment  execution  and  highlight  the 
most  significant  results. 

4.1  Participant  demographics  and  team  composition 

Participants  were  recruited  from  the  graduate  student  population  at  the  NPS.  These  24  subjects  reflected 
a  mean  age  of  32  (standard  deviation  ±  2.7  years),  and  had,  on  average,  1 1  years  of  military  service. 
About  half  the  participants  had  participated  in  previous  A2C2  experiments.  The  subjects  were  grouped  in 
to  teams  of  four,  with  six  teams  total,  or  two  per  each  level  of  the  independent  variable  (Table  3). 


Level  I 

Level  II 

Level  III 

Experimental 

Groups 

Groups  A  and  D 

Groups  B  and  E 

Groups  C  and  F 

Table  3.  Experimental  groups  and  team  assignment. 

We  sought  to  maximize  random  assignment  among  the  teams  to  balance  for  age,  gender,  military 
experience  and  other  exogenous  variables  collected  in  the  initial  demographics.  The  strict  requirement  to 
meet  student  scheduling,  though,  meant  that  Groups  C  and  F  were  somewhat  younger  (team  average  of  29 
years)  and  consequently  less  experienced  as  military  members  (team  average  of  8  years). 

4.2  Critical  task  completion  measures 

Each  four-person  experimental  group  was  further  sub-divided  in  to  two  two-person  geographic  teams, 
each  with  responsibility  for  about  half  of  the  area  of  responsibility  (Figure  1).  Each  geographic  team  was 
given  10  tasks  to  complete  within  the  four  sessions  of  game  play,  with  some  tasks  requiring  careful 
coordination  between  the  teams.  An  example  task  graph  is  given  in  Figure  6. 

Of  these  10  tasks,  several  were  designed  as  “critical”  tasks  with  significant  prerequisites.  These  critical 
tasks  were  designed  to  force  the  experimental  group  to  balance  mission  accomplishment  with  asset 
availability  under  expected  weather  conditions.  The  weather  scenario  was  crafted  intentionally  to 
increase  the  player  workload  and  require  more  intense  deliberation  in  the  completion  of  these  tasks. 

An  overall  completion  score  for  all  tasks  was  computed  for  each  group,  with  scores  ranging  from  88%  to 
95%.  This  overall  measure  represents  the  mean  completion  rate  for  all  tasks  assigned  (Table  4).  To 
examine  the  performance  on  critical  tasks,  a  deviation  score  was  computed  based  on  the  incompleteness 
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of  the  six  critical  tasks,  reflected  as  a  deviation  from  perfect  (Table  4).  This  deviation,  expressed  as  a 
negative  number  in  Table  4,  represents  a  direct  measure  of  the  group  performance  in  critical,  highly 
weather-dependent  tasks,  in  that  a  smaller  (less  negative)  deviation  represents  better  performance. 


Groups 

Overall 

Completion 

Score 

Composite 

Overall 

Completion 

Critical  Task 

Completion 

Deviation 

Composite 
Critical  Task 
Deviation 

Level  I 

A 

93 

91.5 

-23 

-30.5 

D 

91 

-38 

Level  II 

B 

88 

88.5 

-58 

-48.5 

E 

89 

-39 

Level  III 

C 

95 

94.5 

-11 

-15.5 

F 

94 

-20 

Table  4.  Overall  and  critical  task  completion  scores  by  group  and  level. 

A  one-way  analysis  of  variance  on  the  overall  scores  in  Table  4  showed  a  significant  difference  among 
levels  at  p  <  0. 02,  suggesting  that  the  manipulation  of  information  richness  did  indeed  have  an  impact  on 
team  performance.  In  examining  the  critical  task  deviations,  though,  the  differences  among  all  levels  of 
the  independent  variable  drop  to  a  critical  level  of  p  <  0.11.  While  still  significant,  this  suggests  that 
perhaps  some  groups,  regardless  of  level,  may  have  had  some  degree  of  difficulty  in  understanding  and 
integrating  critical  weather  information. 

One  oddity  in  these  data  is  the  relatively  poor  performance  of  the  Level  11  groups  on  critical  and  overall 
task  completion.  These  groups  were  presented  with  a  weather  forecast  with  a  fixed  confidence  interval 
but  no  further  integrating  information.  No  demographic  differences  were  apparent  in  these  groups  (B  and 
E),  and  the  consistency  in  poor  performance  of  both  groups  in  Level  II  suggests  some  inconsistency  in 
this  particular  instantiation  of  the  independent  variable. 

4.3  Efficiency  measures 

While  the  group  measures  of  task  completion  give  broad  insight  into  team  performance,  an  examination 
of  plan  efficiency  among  groups  gives  more  subtle  and  a  more  useful  perspective  on  the  players’ 
application  and  integration  of  environmental  information.  Plan  efficiency  was  measured  in  post¬ 
experiment  analysis  by  comparing  the  daily  plans  produced  within  groups  against  an  ideal  plan  based  on 
perfect  knowledge  of  the  weather.  Monte  Carlo  simulations  were  used  to  identify  the  most  asset-efficient 
task  force  assignment  to  tasks  given  the  current  disposition  of  forces  and  the  current  (true)  weather 
conditions.  Comparisons  of  player  plans  to  these  ideal  plans  showed  scores  from  92  to  100%  (Table  5). 

A  one-way  analysis  of  variance  on  the  data  in  Table  5  shows  significance  at  p  <  0.09,  and  the  direction  of 
the  relation  is  consistent  with  our  experimental  hypotheses.  That  is,  groups  with  richer  information 
appear  to  make  more  efficient  use  of  resources  in  constructing  their  plans.  One  reasonable  inference  from 
these  data  is  that  players  with  richer  information  make  better  global  assignments  of  assets  to  mission 
requirements,  particularly  when  this  information  included  integrated  mission  impact  data  connected  to 
forecast  weather  and  ocean  conditions.  Although  the  p-value  around  0.09  suggests  a  less  powerful  result, 
we  should  keep  in  mind  that  the  teams  in  Level  III  were  both  younger  and  less  experienced  than  their 
counterparts  in  Levels  II  and  I.  Had  we  been  able  to  more  adequately  randomize  the  groups  we  believe 
these  efficiency  measures  would  have  shown  an  even  more  dramatic  difference  among  levels. 
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Groups 

Overall 

Completion 

Score 

Composite 

Overall 

Completion 

Level  I 

A 

92 

93 

D 

94 

Level  II 

B 

99 

98 

E 

97 

Level  III 

C 

100 

99 

F 

98 

Table  5.  Efficiency  measures  of  group  performance. 


4.4  Subjective  measures 

In  addition  to  measures  of  task  completion  and  efficiency,  subjective  measures  were  collected  from  the 
players  at  the  end  of  each  game  session.  These  surveys  were  used  to  assess  cognitive  workload  and  self- 
assessment  of  performance  by  each  of  the  groups. 

Participants  in  the  Level  I  groups  were  split  roughly  50:50  when  asked  if  they  made  changes  from  the  forecast 
regarding  the  weather.  About  half  stated  they  thought  the  weather  would  be  better  and  they  tended  to  be 
optimistic  about  the  weather,  while  half  indicated  they  tended  to  be  pessimistic.  Many  indicated  that  there  were 
not  a  lot  of  data  points  regarding  the  weather,  consistent  with  this  level  of  the  independent  variable. 

Participants  in  the  Level  II  groups  demonstrated  a  consistent  pattern  of  making  changes  to  the  assumed  weather 
from  the  forecast.  They  indicated  they  made  changes  91  percent  of  the  time  and  did  so  for  the  following  reasons: 
because  the  weather  update  indicated  more  favorable  conditions;  because  the  weather  forecast  seemed  overly 
conservative  when  compared  to  the  available  range;  and  because  to  increase  performance  on  task  prerequisites 
that  needed  to  be  accomplished  to  be  better  positioned  for  follow-on  planning  for  tasks.  This  suggests  that  given 
explicit  uncertainty  information,  player  planners  did  try  to  make  good  use  of  this  information. 

Participants  in  the  Level  III  groups  indicated  they  made  changes  from  the  forecast  79%  of  the  time.  Their  reasons 
for  making  changes  were  based  on  updates  that  showed  better  or  worse  than  expected  weather;  recommendations 
from  the  METOC  officer  (an  automated  confederate);  distance  between  ranges  and  the  pattern  of  the  weather;  and 
to  achieve  better  performance  in  terms  of  time  to  complete  tasks.  This  also  suggests  that  given  explicit 
uncertainty  information  as  well  as  mission-impact  information  (e.g.,  thunderstorms  at  48  hours  may  affect  air 
warfare)  player  planners  will  include  this  information  in  their  decision-making  processes. 

Players  were  asked  to  rate  on  a  scale  of  0  to  100  their  perceived  mental  effort;  overall  effort;  time 
pressure;  frustration;  and  performance.  The  composite  means  for  all  players  in  a  group  appear  in  Table  6; 
players  were  surveyed  a  total  of  four  times  over  the  experiment. 

Simple  one-way  analyses  of  variance  on  these  factors  highlight  several  interesting  features  of  the 
experiment.  Both  perceived  mental  effort  and  perceived  overall  effort  show  little  distinction  among 
levels  of  the  independent  variable;  a  reasonable  inference  from  this  evidence  is  that  the  weather  scenario 
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as  designed  did  not  necessarily  favor  the  Level  11  and  Ill  groups.  The  perceived  time  pressure,  though, 
did  show  some  significant  difference  among  groups  (p  <  0.13)  though  the  Level  111  groups  appeared  to 
feel  the  most  pressure.  We  speculate  that  the  relative  youth  and  inexperience  of  this  group  may  have 
contributed  to  a  keener  sense  of  the  time  pressure  in  the  experiment.  This  may  also  have  been  simple 
systematic  error  in  measurement,  though,  as  the  perceived  levels  of  frustration  (p  <  0.57)  and 
performance  (p  <  0.73)  showed  little  distinction  among  the  groups. 


Group 

Mental 

Effort 

Overall 

Effort 

Time 

Pressure 

Frustration 

Performance 

Level  1 

A 

46.7 

40.7 

24.0 

29.7 

87.0 

D 

39.5 

34.0 

25.0 

31.5 

43.0 

Level  II 

B 

40.7 

38.3 

15.0 

33.3 

45.0 

E 

48.0 

42.7 

27.3 

22.3 

93.7 

Level  III 

C 

45.3 

43.3 

36.6 

45.3 

65.0 

F 

44.0 

39.3 

34.4 

28.6 

87.0 

Table  6.  Self-reported  measures  from  post-session  surveys. 

A  multiple  analysis  of  variance  on  these  data  (Table  6)  shows  several  expected  interactions  among 
measures.  In  particular,  perceived  frustration  and  perceived  performance  were  significantly  negatively 
correlated  (r  =  -0.71).  Frustration,  however,  appeared  to  be  relatively  uncorrelated  with  perceived  overall 
effort  (r  =  0.03).  We  speculate  that  frustration,  as  a  measure,  was  more  indicative  of  player  comfort  with 
the  computer-mediated  simulation  rather  than  with  information  delivered  under  different  experimental 
conditions. 

4.5  Summary 

Direct  measures  of  group  performance  support  the  working  hypothesis  that  teams  given  richer 
information  produce  better  and  more  efficient  plans.  The  subjective  measures  collected  during  the 
experiment  suggest  that  the  performance  differences  were  attributable  to  real  differences  among  the  levels 
of  the  independent  variable,  and  not  simple  differences  among  teams  and  team  members.  We  next 
conclude  our  discussion  and  present  several  ideas  for  future  research. 

5.  Discussion 

Does  providing  more  information  to  planners  improve  the  planning  process?  In  this  study  we  have 
investigated  planning  under  uncertainty,  manipulating  information  richness  as  the  independent  variable. 
Human  subjects,  playing  the  role  of  Maritime  Operations  Center  planners,  participated  in  an  eight-hour 
computer-mediated  game  in  a  scenario  where  successful  mission  accomplishment  required  players 
evaluate  and  integrate  expected  weather  information.  Experimental  results  suggest  that  groups  operating 
with  richer  information  (weather  forecasts  with  explicit  uncertainty  and  integrated  with  mission  impacts) 
significantly  outperform  planning  teams  operating  with  less  rich  information  (simple  deterministic 
forecasts). 

Recent  trends  in  both  the  Navy  and  Air  Force  provide  significant  motivation  for  this  study.  The 
increasing  environmental  sensitivity  of  weapons  systems,  particularly  autonomous  systems,  places  a 
higher  premium  on  weather  and  ocean  information  in  defense  operations.  The  state  of  the  science  in 
atmospheric  and  ocean  prediction  requires  a  blend  of  human  and  automated  processing  to  create 
actionable  intelligence  from  imperfect  numerical  forecasts  (Doswell,  2004;  Lorenz,  1963).  Although  the 
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focus  of  this  study  was  on  planning  under  uncertainty,  we  can  think  of  the  levels  of  the  independent 
variable  (information  richness)  as  representing  different  combinations  of  human  and  automated 
processing.  Future  investigations  could  refine  and  more  deeply  examine  the  optimal  mix  of  human  and 
automation  within  the  meteorological  and  oceanographic  (METOC)  communities. 

In  this  study,  planning  under  uncertainty  focused  on  METOC  data  as  the  source  of  uncertain  information. 
A  future  investigation  might  consider  the  more  general  case  of  highly  uncertain  intelligence  in 
constructing  operational  plans.  Sources  of  uncertainty  in  these  studies  might  range  from  uncertainty  in 
modeling  the  physical  atmosphere  and  oceans,  to  estimates  of  enemy  capabilities  and  intentions.  Such  a 
study  would  better  inform  applied  or  operational  research  in  human-to-machine  interfaces  in  command 
and  control  systems. 

We  have  established  in  this  work  that  providing  richer  information  (explicit  uncertainty)  to  planners 
enables  planners  to  deliver  more  effective  and  efficient  plans.  We  sought  to  model  as  closely  as  possible 
the  planning  processes  inside  the  Maritime  Operations  Center  (MOC).  Future  investigations  would  refine 
this  model  of  the  MOC,  and  extend  this  work  into  the  Air  Operations  Center  (AOC)  and  Joint  Operations 
Center  (JOC).  Among  the  operational  forces  today,  the  proliferation  of  information  (command  and 
control)  systems  appears  to  put  a  wealth  of  data  in  the  hands  of  planners,  though  not  all  of  these  data  may 
be  immediately  useful  or  relevant.  In  this  work,  we  have  sought  to  better  understand  how  these  data 
become  information  in  the  human  process  of  command  and  control  with  the  addition  of  explicit 
uncertainty  bounds  on  these  data.  We  expect  these  results  will  inform  further  studies  to  improve 
understanding  of  the  best  mix  of  human  and  automated  processing  that  leads  to  better  performing  teams 
and,  ultimately,  more  effective  planning. 
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Background 

A2C2  Research  Program 


•  Adaptive  Architectures  for  Command  and  Control  (A2C2)  has  operated  as 
a  research  program  at  Naval  Postgraduate  School  for  over  15  years 

-  Integrates  analytic  modeling,  human-in-the-loop  experimentation  and 
simulation  in  a  research  paradigm  of  model-test-model-experiment 

-  Models  and  associated  simulations  define  and  guide  the  experiments, 
and  results  from  experiments  are  then  used  to  improve  models. 

•  Over  the  past  three  years,  A2C2  investigators  have  developed  a  multi¬ 
disciplinary  research  agenda  to  explore  issues  critical  to  the  Maritime 
Operations  Centers  (MOC). 


Operational  planners  are  often  faced  with  constructing  robust,  effective 
plans  using  ambiguous  information  in  a  complex  or  evolving  situation. 


Environmental  information  represents  a  particular  challenge  for  the 
planner— weather  and  ocean  forecasts  carry  significant  uncertainty. 


Even  with  perfect  knowledge,  translating  these  conditions  into  mission 
impacts  can  be  difficult -delivering  a  more  accurate  weather  forecast  does 
not  necessarily  provide  the  planners  with  more  useful  information. 


“ Listen ,  S-2,  "  the  colonel  said,  “l  don  ’t  care  about  how  many  inches  of  rainfall  to  expect.  I  don  ’t  care 
about  the  percentage  of  lunar  illumination.  I  don  ’t  want  lots  of  facts  and  figures.  Number  one,  I 
don  ’t  have  time,  and  number  two,  they  don  ’t  do  me  any  good.  What  I  need  is  to  know  what  it  all 
means.  ” 

— USMC  Doctrinal  Publication  6  Command  and  Control  (1996) 
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Background 

Weather  and  Uncertainty 


•  Weather  prediction  is  appealing  as  a  purely  deterministic  problem— but 
even  the  current  state  of  the  science  shows  limited  skill  beyond  six  days 

•  Every  numerical  weather  prediction  carries  an  inherent  uncertainty. 

-  Typically,  the  bounds  of  this  uncertainty  are  unknown,  though  a  skilled  human  weather 
forecaster  can  attach  a  reasonable  estimate  of  the  uncertainty  to  a  forecast. 

-  Although  this  uncertainty  estimate  may  be  qualitative  the  human  forecaster 
nonetheless  can  make  this  uncertainty  clearer  to  decision-makers. 

•  Ensemble  forecasting  is  an  explicit  approach  to  evaluate  model 
uncertainty,  comparing  several  models  over  the  same  forecast  period 

-  The  explicit  means  and  variances  derivable  from  ensemble  output  provide  another 
means  to  make  clear  to  decision-makers  how  much  trust  to  place  in  a  particular  forecast 

-  Interpretation  of  these  numbers however ,  is  not  always  clear  to  decision-makers 
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Background 

Weather  and  Decision-making 


•  No  matter  the  skill  with  which  we  predict  the  natural  environment,  these 
predictions  are  of  themselves  little  use  to  military  planners. 


•  Decision-makers  are  largely  concerned  with  when  and  to  what  degree 
their  assets  and  capabilities  will  be  affected  by  weather  conditions 

•  For  planners  with  a  trade-space  spanning  days  or  weeks,  decisions  to 
proceed  with,  accelerate  or  delay  operations  are  connected  to  the 
expected  atmospheric  and  ocean  conditions. 

•  Effective  planning  in  this  case  is  connected  to  knowledge  and  exploitation 
of  the  natural  environment. 
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Research  Questions 


•  We  seek  to  better  understand  how  organizations  can  employ  perishable 
and  uncertain  information  in  the  operational  planning  process. 


-  In  the  face  of  inherently  uncertain  information,  if  we  make  this 
uncertainty  more  clear  through  human  intercession,  or  explicit 
quantitative  bounds,  how  will  planners  apply  this  information? 


-  Given  actionable  mission  impacts  connected  to  this  uncertain 

information,  how  will  planners  integrate  and  apply  this  information? 

•  In  the  context  of  the  Maritime  Operations  Center,  does  providing  richer 
information  to  planners  lead  to  better  planning? 
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Experimental  Design 

Modeling  Formalism:  Matching  Assets  to  Tasks 


uu| 


Objective:  Break  area  denial  that  has  been 
established  by  Country  Red  as  it  tries  to  extend  its 
influence  over  Country  Brown  by  force. 


Secondary  considerations:  Allies  in  Country  Green 
must  be  defended  from  any  action  by  Red  or  Brown 


400 
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Experimental  Method 

Modeling  Formalism:  Matching  Assets  to  Tasks 


Area  A  Tasks 
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Experimental  Method 

Modeling  Formalism:  Matching  Assets  to  Tasks 


TASKS  AND  ASSETS  ARE 
REFERENCED  TO  THE  SAME 
SET  OF  SELECTED  WARFARE 
CATEGORIES 

•  Rik:  Requirement  k  of  task  / 

•  Tjk:  Capability  k  of  asset/TF  j 


Values  (resource  vectors)  are 
readily  obtained  via  discussions 
with  SMEs  or  Fleet  personnel 


Future  operations  (FOPS)  planners  assign  each  task  to  a  task  force  (TF) 

-  Planner  assignments  (requests)  include  performance  goals  and  priorities 

-  One  task  force  is  designated  as  primary  by  the  FOPS  planner 

-  Planner  may  assign  other  TFs  as  supporting  in  one  or  more  warfare  area 

-  The  task  forces  (computer  agents)  determine  how  to  best  use  assets  to  meet  goals 
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Experimental  Method 

Laboratory  Environment 


_?  J 

3  J 
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FOPS  PLANNING  TEAM 


Agent 
Algorithm 


_ _ 

FOPS 

PLANNING 

SOFTWARE 

Assignment  ^ 

FOPS 

Network 

SERVER 

A  Ik 

Plan  for  Day  X 
^  Expected 

o 

Database 

performance 

^ - ' 

•  Static  (task  and  asset)  data 

•  Scenario  information 

•  Dynamic  information 

•  Automated  data  collection 


Plan  Summary  T  T+l  T+2 
Asset  Status  T  T+l  T+2 


Players  produce  a  “plan”  by  assigning  task  forces  to  all  active 


Task  Assignment  T  T+l  T+2 

V 

Web  pages 

_ / 

2 


tasks  on  day  T+l  and  T+2,  then  UCONN  agent  algorithms 
allocate  TF  assets  to  best  meet  overall  task  performance  goals 


Each  team  member  has  a  different  planning  responsibility: 
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The  team  develops  a  plan  by  considering 
the  critical  task  prerequisites;  planned  task 
start  dates;  and  the  weather  forecasts  for 
each  area  of  responsibility. 


Trial  or  working  plans  can  be  submitted  to 
the  task  forces  for  review.  This  submission 
returns  an  expected  performance  for  the 
given  plan,  based  on  assets  available  and 
the  weather  impacts  to  those  assets.  This 
evaluation  is  computed  by  the  agent-based 
model. 

The  team  then  modifies  assignments  on 
those  tasks  not  meeting  desired  criteria 

When  the  team  believes  the  plan  is 
satisfactory,  the  plan  is  finalized  so  that: 
T+l  plan  =>  EXORD  for  tomorrow 

f  mill 

T+2  plan  =>  start  for  next  T+l  plan 


Experimental  Design 

Player  Tasks 


Area  A  Area  B 

T+l  FOPS  1  FOPS  3 

Sequential 
Coordination 

T+2  FOPS  2  FOPS  4 

Parallel  Coordination 


Over  the  four  2-hour  sessions,  players  should  trade  off 
assets  from  tasks  in  bad  weather  areas  to  tasks  in 
good  weather  areas,  and  defer  starting  tasks  from  T+l 
to  T+2  (or  longer,  within  commander’ s  guidance)  if 
the  weather  is  forecast  to  improve. 
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Independent  Variables 


Level  1 

Level  II 

Level  III 

Information 

Richness 

Forecast  only,  no 
additional  weather  or 
mission  information 

Forecast  with  explicit 
uncertainty  information 
but  no  additional 
mission  information 

Forecast  with  explicit 
uncertainty  and 
integrating  mission 
information 

To  better  meet  our  experimental  objectives  we  chose  to  examine  both 
aspects  of  the  weather  specialists’  contribution  to  information 
processing  using  a  compromise  design. 
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Experimental  Design 

Operationalizing  the  independent  variable 


File  Help 

Refresh 

Get  DB  Data  (n  Start Exp  | 

Summary  Task  Status  Asset  Status  Assignment 

DAYT  DAY  T  +  1  |  DAY  T  +  2  | 


Weather 


SELECTED  AREA 


r  a 


^  B 


UPDATE 


Area  B  Forecast  T+1:  Expect  strong  thunderstorms  to  the  south  of  Area  B.  with  heavy  seas  throughout  the  region  and  cloud  cover 
affecting  operations  through  the  next  two  days  at  least 


I 


Forecasted 
Sea  Conditions 


55 


Forecasted 
_  £[omL,C(ffpr. 


40 


Forecasted 

T'STORMS 


J,  > 


Sea  Condition 

r3 

Cloud  Cover 

T'storms 

l“u  JJ 

%EFFECTIVENESS  OF  ASSET  CAPABILITIES  IN  THIS  WEATHER 

APPLY 

Forecast 
Uncertainty 
(Level  II,  III) 


What-if 
Evaluation 

(Level  I,  II,  I II  V'ssumed  -5 


v.M.li 


Explicit  Mission 
Impacts  (Level 


%  OF  PERFECT  WEATHER  PERFORMANCE  LIKELY  OBTAINABLE  IN  THIS  WEATHER 


Task  ID 

TB01 

TB02 

TB03 

TB04 

TB05 

TB06 

TB07 

TB08 

%Perf 

59 

53 

70  1 

?? 

?? 

?? 

?? 

?? 

TB09 


TB10 
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Dependent  Measures 


•  Broadly,  we  hypothesize  that  planning  teams  given  richer  environmental 
information  will  perform  better.  “Better”  in  this  sense  means  that  teams 
made  better  use  of  limited  assets,  and,  ultimately,  produced  better  plans 

•  Completion  measures  are  used  to  evaluate  team  performance  in 
completing  the  assigned  tasks  within  their  task  graphs 

-  We  hypothesize  that  teams  in  Level  III  (integrative  weather  and  mission  impact 
information)  will  outperform  teams  in  Levels  I  and  II. 

•  Efficiency  measures  examine  how  well  a  team  applied  assets  to  tasks. 

-  Indicates  how  well  a  team  adjusted  to  bad  weather  by  shifting  work  from  between 
areas 

-  We  hypothesize  that  teams  in  Level  II  and  III  will  make  more  efficient  use  of  assets  than 
teams  in  Level  I.  We  expect,  too,  that  Level  III  will  be  more  efficient  than  Level  II. 
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Experimental  Results 

Completion  Measures 


Groups 

Overall 

Completion 

Score 

Composite 

Overall 

Completion 

Critical  Task 

Completion 

Deviation 

Composite 
Critical  Task 
Deviation 

Level  I 

A 

93 

91.5 

-23 

-30.5 

D 

91 

-38 

Level  II 

B 

88 

88.5 

-58 

-48.5 

E 

89 

-39 

Level  III 

C 

95 

94.5 

-11 

-15.5 

F 

94 

-20 

•  One-way  ANOVA  shows  a  significant  difference  at  p  <  0.02,  suggesting 
that  manipulation  of  information  richness  did  impact  team  performance. 


•  For  critical  task  deviations  differences  among  all  levels  of  drop  to  a 
critical  level  of  p  <  0.11  ...  groups  may  have  had  trouble  understanding 
the  task  graphs  and  the  scenario  critical  path 
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Experimental  Results 


Efficiency  Measures 


Groups 

Efficiency 

Score 

Composite 

Efficiency  Score 

Level  I 

A 

92 

93 

D 

94 

Level  II 

B 

99 

98 

E 

97 

Level 

III 

C 

100 

99 

F 

98 

•  One-way  ANOVA  shows  significance  at  p  <  0.09,  and  the  direction  of 
the  relation  is  consistent  with  our  experimental  hypotheses. 

•  One  inference  from  these  data  is  that  players  with  richer  information 
make  better  global  assignments  of  assets  to  mission  requirements, 
particularly  when  this  information  included  integrated  mission  impact 
data  connected  to  forecast  weather  and  ocean  conditions. 
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Experimental  Results 


Survey  Measures 


Group 

Mental 

Effort 

Overall 

Effort 

Time 

Pressure 

Frustration 

Performance 

Level  I 

A 

46.7 

40.7 

24.0 

29.7 

87.0 

D 

39.5 

34.0 

25.0 

31.5 

43.0 

Level  II 

B 

40.7 

38.3 

15.0 

33.3 

45.0 

E 

48.0 

42.7 

27.3 

22.3 

93.7 

Level  III 

C 

45.3 

43.3 

36.6 

45.3 

65.0 

F 

44.0 

39.3 

34.4 

28.6 

87.0 

•  MANOVA  shows  several  interesting  interactions  among  measures. 

•  Perceived  frustration  and  perceived  performance  were  significantly 
negatively  correlated  (r  =  -0.71) 

•  Frustration,  however,  appeared  to  be  relatively  uncorrelated  with 
perceived  overall  effort  (r  =  0.03) 

•  We  speculate  that  frustration,  as  a  measure,  was  more  indicative  of 
player  comfort  with  the  computer-mediated  simulation  rather  than  with 
information  delivered  under  different  experimental  conditions. 
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Conclusions 

Operational  relevance 


•  We  expect  that  teams  given  richer  information  will  engage  in  more 
effective  planning  and  likely  will  produce  a  better  plan  ...  so  what? 

•  Our  design  is  intended  to  address  the  more  useful  question:  how  much 
better  do  teams  perform  given  richer  uncertainty  information? 

•  This  question  is  of  significant  operational  relevance  to  both  the  Navy  and 
Air  Force,  as  there  is  a  cost  to  keep  humans  deeply  embedded  in  the 
forecast  process,  and  a  cost  to  produce  explicit  uncertainty  bounds  with 
numerical  forecasts. 
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Conclusions 

Operational  relevance 


•  Within  the  DoD  the  current  trend  is  to  consolidate  METOC  personnel  in 
centers  located  far  from  the  forward  edge  of  battle,  and  most  often 
located  in  the  CONUS.  Support  to  deployed  operations  is  then  provided 
with  online  product  delivery  and  reach  back  service  to  these  centers. 


•  Both  the  Navy  and  the  Air  Force  are  examining  the  use  of  ensemble 
numerical  weather  prediction  to  improve  operational  forecasts  and 
improve  the  explicit  uncertainty  information  attached  to  these  forecasts. 

•  For  both  services  a  lingering  concern  is  whether  decision-makers  will 
correctly  and  effectively  employ  this  richer  information— insights  from 
this  study  may  prove  useful  to  Navy  and  Air  Force  organizations  shaping 
and  re-shaping  their  decision-support  and  planning  processes. 
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Conclusions 

The  way  ahead 


•  This  work  motivates  several  avenues  for  future  investigation,  including: 


-  Planning  under  high  task  uncertainty  (e.g.  reconnaissance,  close  air 
support,  casualty  evacuation)  interacting  with  uncertainty  in  the 
natural  environment 


-  Measures  of  trust  in  weather  forecasts  and  other  intelligence 
products 

-  Human  factors  in  C2  systems:  creating  actionable  intelligence  from 
multiple  sources 


-  Active  and  passive  deception  detection  in  C2  planning  systems 


NAVAL 

POSTGRADUATE 

SCHOOL 


Backup  Slides 


Monterey,  California 

WWW.NP^EDU 


NAVAL 

POSTGRADUATE 

SCHOOL 


Experimental  Design 

Modeling  Formalism:  Matching  Assets  to  Tasks 


File  Help  Current  Time  Period  [o 

Summary  Task  Status 

i  DAyT]  DAYT  +  1  |  DAY T ♦  2  | 


Assignment 


Task  assignment  page 


|  TASK:  | 


|  DESIRED  [%  COMPLETEf 


|  TAMP  GREEN  (TA02) 

1  65  - 


TASK  PRIORITY 


RESPONSIBLE  DM: 
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0 
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r.l* 

Plan  is  submitted  to  “TFs”  for  review 

•  FOPS  assesses  expected  performance 

•  Modifies  assignments  on  those  tasks  not 
meeting  desired  criteria 

•  When  satisfactory,  the  plan  is  “finalized” 

-  T+2  plan  =>  start  for  next  T+l  plan 

-  T+l  plan  =>  EXORD  for  tomorrow 


Rolling  Horizon 
Planning 


For  each  task,  the  responsible 
FOPS  planner  assigns: 

1)  one  primary  TF 

2)  up  to  two  supporting  TFs 

•  each  in  up  to  2  warfare  areas 

3)  desired  perf  level  (accuracy,  %  complete) 
The  plan  =  aggregate  of  all  assigned  tasks  for 
the  given  day,  is  posted  on  the  summary 


Cuironl  lime  Ponoo  0 

Summary 

Lamad  oavt*i 


Plan  summary  page 
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Experimental  Design 

Plan  of  execution 


The  experiment  occurs  in  five  two-hour  time  blocks: 

-  Block  0  is  an  introduction  to  the  experiment,  including  a  brief  on  the  mission,  and  initial 
training  for  players  in  their  roles.  The  MOC  Director  (a  confederate)  leads  the  session. 

-  Block  1  is  the  first  full  session  where  players  are  given  the  initial  state  of  the  scenario 
and  are  tasked  with  building  plans  for  Block  2  (T+l)  and  Block  3  (T+2).  These  plans  will 
be  briefed  to  the  MOC  Director  and  submitted  as  a  new  plan  at  the  end  of  this  block. 

-  Block  2  begins  with  the  implementation  of  the  FOPS  team  plan  produced  in  Block  1. 

The  FOPS  team  will  create  a  plan  for  Block  3  using  their  Block  1  plan  for  T+2  as  a  first 
guess.  In  this  block,  teams  will  also  create  the  Block  4  (T+2)  plan  from  the  updated 
Block  3  plan. 

-  Block  3  implements  the  Block  2  plan  for  T+l,  and  teams  use  the  Block  2  T+2  plan  as  the 
starting  point  for  the  T+l  (Block  4)  plan.  At  the  end  of  this  session,  another  update 
briefing  will  be  given  to  the  MOC  Director. 

-  Block  4  is  the  final  session  of  the  experiment.  This  session  implements  the  Block  3  plan 
for  T+l,  and  teams  will  use  the  Block  3  T+2  plan  as  the  starting  point  for  the  T+l  (Block 
5)  plan.  Expected  progress  will  guide  the  T+2  (Block  6)  plan,  though  neither  the  T+l  nor 
the  T+2  plan  will  be  executed. 
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Experimental  Design 

Independent  Variables 


Information  Content  (a,b) 

Forecast  Only 

Forecast  with 
Explicit  Uncertainty 

Information 

Structure 

(UD 

Automated  Products 

Level  la 

Level  lb 

Automated  Products  with 
Human  Expertise 

Level  lla 

Level  Mb 

The  levels  of  the  independent  variable  as  originally  conceived  separated  content  and 
structure  of  the  forecast  products  into  a  classic  2X2  design. 

Levels  la  and  I  la  represent  much  of  the  current  practice  in  the  Navy,  Air  Force  and  National 
Weather  Service.  Operationally,  most  atmosphere  and  ocean  products  are  presented  as 
deterministic  forecasts  with  implicit  uncertainty  (la).  This  uncertainty  is  often  clarified  by 
additional  information  from  experienced  human  forecasters  (lla).  The  Navy  and  Air  Force 
are  both  considering  moving  to  product  portfolios  with  ensemble  products  (Level  lb  or  Mb). 
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